1. Field of the Invention
The present invention relates to an image processing method, an image processing apparatus for use with a video camera, a digital still camera and a silver halide camera and an image pickup apparatus and a display apparatus suitable for the application of this image processing method.
2. Description of the Related Art
So far distortion has been caused in an image picked up by a video camera, a digital still camera or a silver halide camera by influences of distortion characteristics of an image pickup lens. While such distortions are inconspicuous in highly-accurate and high-performance lenses, when an inexpensive lens or an optical zoom lens is in use, it is difficult to avoid influences exerted by distortions completely.
To avoid these defects, in recent years, an image processing apparatus has been proposed to correct distortions by signal processing.
FIG. 1 of the accompanying drawings is a block diagram showing an arrangement of an image processing apparatus 100 according to the related art. As shown in FIG. 1, the image processing apparatus 100 according to the related art includes a lens 200, an image pickup element 300, a data converting unit 400, a signal processing unit 500, an image memory 600, a control microcomputer 700, a synchronizing signal generating unit 800, a correction data table 1000, a recording unit 1100, a reproducing unit 1200 and a display-system processing unit 1300.
Outlines of operations of the above-described image processing apparatus 100 will be described with reference to a flowchart of FIG. 2.
Referring to FIG. 2, and following the start of operation, control goes to a step S1, whereat an analog image signal for an object 100 is inputted to the image processing apparatus 100 through the lens 200 and the image pickup element 300. Control goes to the next step S2, whereat the data converting unit 400 converts the analog image signal to a digital image signal to thereby generate an image 102.
In the next step S3, the signal processing unit 500 effects correction calculation on the distorted image 102 by using a distortion correction vector (hereinafter simply referred to as a “correction vector”) stored in the correction data table 1000. In the next decision step S4, it is determined by the control microcomputer 700 whether or not inputting of the analog image signal should be ended. If inputting of the analog image signal is not ended as represented by a NO at the decision step S4, control goes back to the step S1. If a YES is outputted at the decision step S4, control is ended.
The outline of the operations of the related-art image processing apparatus 100 has been described so far, and those operations will be described below more fully.
Referring back to FIG. 1, the lens 200 focuses reflected light from an object 101 and maps the focused reflected light on the image pickup element 300. The image pickup element 300 is composed of a suitable device such as a CCD (charge-coupled device) and a CMOS (complementary metal oxide semiconductor) sensor, and captures the mapped image to generate the analog image signal. The data converting unit 400 converts the analog image signal supplied from the image pickup element 300 to provide the digital image signal to thereby generate the image 102. On the other hand, the control microcomputer 700 issues a command for instructing a predetermined operation in response to data inputted to an external user interface.
The signal processing unit 500 stores the digital image signal generated from the data converting unit 400 in the image memory 600 in response to the command supplied thereto from the control microcomputer 700. Then, the signal processing unit 500 reads out correction vectors corresponding to all pixels from the correction data table 1000 and obtains a necessary image signal from the image memory 600 in response to the correction information. Then, the signal processing unit 500 corrects a distortion of the image 102 outputted from the data converting unit 400 by effecting two-dimensional interpolation system geometric correction on the image signal.
The image signal generated by the signal processing unit 500 is supplied to the display-system processing unit 1300 and thereby the image 103 is displayed on a monitor (not shown). Alternatively, the image signal generated by the signal processing unit 500 is supplied to the recording unit 1100 and thereby the image is recorded on a media 1400 such as an external magnetic tape or a disc or a memory. The image signal recorded on the media 1400 is reproduced by the reproducing unit 1200 and the reproduced signal is supplied to the display-system processing unit 1300, whereby the reproduced image 103 is displayed on the monitor.
Further, the synchronizing signal generating unit 1600 generates an internal synchronizing signal in response to a clock signal CLK supplied thereto from the outside and supplies the thus generated internal synchronizing signal to the image pickup element 300, the data converting unit 400 and the signal processing unit 500.
In this manner, the distortion can be corrected by signal processing. In the technical fields of similar kind, the following cited references have been laid open.
[Cited Patent Reference 1]
Japanese laid-open patent application No. 5-172531
[Cited Patent Reference 2]
Japanese laid-open patent application No. 6-165024
[Cited Patent Reference 3]
Japanese laid-open patent application No. 6-181530
The assignee of the present application has previously proposed Japanese patent application No. 2002-239865. FIG. 3 shows an arrangement of an image processing system that the assignee of the present application has previously proposed (Japanese patent application No. 2002-239865). In FIG. 3, elements and parts identical to those of FIG. 1 are denoted by identical reference numerals.
As shown in FIG. 3, the image processing system according to the previously-proposed patent application comprises the image processing apparatus 100, a pre-processing apparatus 1500 and the media 1400. The image processing apparatus 100 includes the lens 200, the image pickup element 300, the data converting unit 400, the signal processing unit 500, the image memory 600, the control microcomputer 700, a correction parameter decoder 1010, the synchronizing signal generating unit 1600, the recording unit 1100, the reproducing unit 1200 and the display-system processing unit 1300. The pre-processing apparatus 1500 includes a correction parameter encoder 900 and a correction parameter deriving unit 800.
The lens 200 focuses reflected light from the object 101 and maps the focused reflected light on the image pickup element 300. The lens 200 is not limited to a single focal point lens and may be a lens with zoom function. The image pickup element 300 is composed of a suitable means such as a CCD and a CMOS sensor and captures a mapped image in response to the internal synchronizing signal supplied thereto from the synchronizing signal generating unit 1600 to generate an analog image signal.
The data converting unit 400 is connected to the image pickup element 300 and converts the analog image signal generated from the image pickup element 300 in response to the internal synchronizing signal supplied thereto from the synchronizing signal generating unit 1600 into a digital image signal to thereby generate the image 102. The signal processing unit 500 is connected to the control microcomputer 700, the data converting unit 400, the image memory 600, the correction parameter decoder 1010 and the synchronizing signal generating unit 1600.
Then, the signal processing unit 500 stores the digital image signal supplied thereto from the data converting unit 400 in the image memory 600 in response to the command supplied from the control microcomputer 700 and also effects correction processing on the stored image signal based on the correction amount parameter supplied from the correction parameter decoder 1010. Then, the signal processing unit 500 supplies an image signal obtained by the above correction to the display-system processing unit 1300 and the recording unit 1100.
On the other hand, the correction parameter deriving unit 800 calculates in advance correction amount vectors corresponding to respective positions of all pixels from data concerning distortion of the lens 200. The correction parameter encoder 900 is connected to the correction parameter deriving unit 800 and the user interface and compresses (encodes) the correction amount vectors supplied from the correction parameter deriving unit 800 in response control signals Ln, Lw supplied from the user interface and supplies compressed data Pc to the correction parameter decoder 1010.
Although calculation and encoding in the correction parameter deriving unit 800 become calculations having very large calculation loads, such calculations may be executed separately by other suitable means such as a personal computer, and hence may not exert an influence upon real-time processing done by the image processing apparatus 100. Information such as the compressed data Pc supplied from the correction parameter encoder 900 is also stored in the distortion correction memory 1020 so that information stored in the distortion correction memory 1020 is used in the processing made in actual practice by the correction parameter decoder 1010.
Further, the control microcomputer 700 supplies a command for instructing predetermined operations and the like to the signal processing unit 500 in response to the control signal from the user interface and also supplies position information of the lens 200 to the correction parameter decoder 1010.
The correction parameter decoder 1010 is connected to the correction parameter encoder 900, the distortion correction memory 1020, the control microcomputer 700 and the signal processing unit 500. Then, in response to the information supplied from the control microcomputer 700 and the like, the correction parameter decoder 1010 expands (decodes) the thus encoded compressed data Pc supplied from the correction parameter encoder 900 or the distortion correction memory 1020 to provide correction amount parameters corresponding to respective pixels and supplies the correction parameters to the signal processing unit 500.
The recording unit 1100 is connected to the signal processing unit 500 and records the image signal generated from the signal processing unit 500 on the media (recording medium) 1400 such as a magnetic tape, a flexible disc, a DVD (digital versatile disc), a hard disk and a memory. Alternatively, the image signal generated by the signal processing unit 500 may be recorded on the media 1400 by using a suitable means such as the Internet and wireless communication.
The reproducing unit 1200 is connected to the media 1400 and reproduces the image signal from the media 1400, thereby supplying the reproduced image signal to the display-system processing unit 1300. The display-system processing unit 1300 is connected to the signal processing unit 500 and the reproducing unit 1200 and displays the image signal supplied from the signal processing unit 500 or the reproducing unit 1200 to the monitor (not shown).
The synchronizing signal generating unit 800 generates an internal synchronizing signal in response to a clock signal CLK supplied thereto from the outside and supplies the thus generated internal synchronizing signal to the image pickup element 300, the data converting unit 400 and the signal processing unit 500.
Further, the details of the correction parameter encoder 900 will be described below. As shown in FIG. 4, a lattice dividing unit 901 determines a dividing method of distortion correction coordinates of the whole picture. A polynomial of degree n coefficient deriving unit 902 expresses all distortion correction coordinates of respective lattice segments divided by the lattice dividing unit 901 by function with the position on the lattice segment and approximates all distortion correction coordinates by target division polynomial of degree n. The distortion correction parameters are data retained in order to carry out distortion correction processing.
Polynomial of degree n is an equation expressed as:F(x)=a(n)*xn+a(n−1)*x(n−1)+a(n−2)*x(n−2) . . . +a(0)
The division polynomial of degree n is to express only an interval divided by a certain effective range with polynomial of degree n. The lattice segment represents a lattice segment divided by lattice segments (segment between lattice points).
For reference, the dividing method will be described below with reference to FIGS. 5A, 5B and FIG. 6. Specifically, when the lattice is divided, there are available a method of using the whole picture as a picture dividing target as shown in FIG. 5A and a method of using only ¼ of picture, as shown in FIG. 5B, because the distortion of lens has point symmetry. These two methods may be selected based upon distortion characteristics.
As a method of dividing the distortion correction coordinates into lattices, mainly three methods are proposed. Specifically, the equal dividing method of the first method is to equally divide the distance from the center to the end by the number of divided lattices as shown on the upper right of FIG. 6. The dividing method using power of 2 of the second method is to divide the distance from the center to the end at every power of 2 as shown on the middle of FIG. 6. This dividing method using power of 2 is able to alleviate calculation processing.
Further, the optimum value dividing method of the third method is to use distortion correction coordinates of the upper end and right end of the picture where distortions of figure shown on the upper left in FIG. 6 are largest to create function h(x) in which Y axis represents the distortion correction coordinates and X axis represents the distance from the center as shown on the lower right of FIG. 6. Then, by using the division polynomial of degree n, a dividing position in which an error between the function h(x) and the division polynomial of degree n may be minimized is searched for.
With respect to the method of calculating the polynomial of degree n, FIGS. 7A to 7D show an example of the division polynomial of degree n for reference. Specifically, as shown in FIG. 7A, one lattice segment is selected. Here, assuming that x1=x0+1, a division polynomial of degree 2 that passes through three points of (x0, h(x0)), (x1, h(x1)), (x2, h(x2)) is calculated. As shown in FIGS. 7B, 7C, between both ends x0, x2 of the selected lattice segment, x1 is shifted one by one from x0 to x2 and the above-described calculation is carried out until the point of x2−1. In this manner, the division polynomial of degree 2 of which the error relative to the original function is smallest is calculated from all division two-dimensional polynomial is calculated. Further, the above-mentioned processing is carried out with respect to all lattice segments shown in FIG. 7D.
FIG. 8 is a flowchart showing operations of the parameter encoder 900. Referring to FIG. 8, and following the start of operation, control goes to a step S11, whereat distortion correction coordinates relative to all pictures on the target picture are read out. In the next step S12, the picture is divided into lattices in accordance with the lattice dividing method. Then, control goes to a step S14, whereat one lattice segment encircled by lattice points is selected.
Control goes to the next step S15, whereat y axis represents all distortion correction coordinates on the selected lattice segment and x axis represents the distance on the lattice line to create a discrete function, and the discrete function is approximated by the division polynomial of degree n. Further, at a step S18, n coefficients of division polynomial of degree n are transferred to the distortion correction memory 1020 as decoded distortion correction parameters. Then, control goes the next decision step S19, whereat it is determined whether or not deriving of distortion correction parameters with respect to all lattice segments is ended. If deriving of distortion correction parameters is not yet finished as represented by a NO at the decision step S19, then control goes back to the step S14. If on the other hand deriving of distortion correction parameters is finished as represented by a YES at the decision step S19, then control is ended.
The coefficients of the division polynomial of degree n that is the thus calculated approximate expression are stored in the distortion correction memory 1020 as distortion correction parameters. The lattice information is stored in the lattice information buffer 1001 disposed within the correction parameter decoder 1000.
Further, the correction parameter decoder 1000 is the block that decodes the distortion correction coordinates of respective points of the picture by the distortion correction parameters obtained from the correction parameter encoder 900 in accordance with the request from the signal processing unit 500. The correction parameter decoder 1000 will be described in detail with reference to FIG. 9.
As shown in FIG. 9, lattice information indicative of division positions of X,Y directions are inputted to the correction parameter decoder 1000 from the correction parameter encoder 900. The inputted lattice information is stored in the lattice information buffer 1001. Then, when the correction parameter decoder 1000 receives request of coordinates (xt, yt) which should be corrected in distortion from the signal processing unit 500, the lattice determining unit 1002 compares lattices containing target coordinates (xt, yt) with lattice position information of the lattice information buffer 1001 and calculates the compared result.
Further, the lattice determining unit 1002 generates addresses at which the respective coefficients of the division polynomial of degree n that are four lattice segments encircling the calculated lattice and requests the distortion correction memory 1020 to read the thus generated addresses. At the same time, the lattice determining unit 1002 supplies the determined lattice position information to the normalizing unit 1003. The normalizing unit 1003 calculates relative coordinates (xt′, yt′) used in the four lines of the encircled lattice from the target coordinates (xt, yt), the lattice positions and lattice information.
The function converting unit 1005 converts the coefficient of the function received from the distortion correction memory 1020 into the form of the function that is used in the plane interpolation. Further, the plane interpolation unit 1006 creates a plane function by using the plane interpolation obtained by the function converting unit 1005 and obtains respective coordinates from the normalizing unit 1003 to thereby calculate desired distortion correction coordinates. Then, calculated results are supplied to the signal processing unit 500.
Accordingly, the flowchart of the correction parameter decoder 1000 is presented as shown in FIG. 10. Referring to FIG. 10, and following the start of operation, at a step S21, lattice information is read as initialization. Further, at a step S22, the signal processing unit 500 is placed in the standby mode (WAIT) until the distortion request signal is issued.
When it is requested at the step S22 to decode the distortion correction parameters, the distortion correction memory 1020 receives coordinates of a target point from the signal processing unit 500. Further, at a step S23, a lattice containing the coordinates of the target point is determined. At a step S24, relative coordinates are calculated from lattice lines of the target point. At the next step S25, one line amount is selected from four line amounts encircled as the lattice and distortion correction parameters (coefficients) of the division polynomial of degree n are read.
If processing concerning the four line amount is not yet ended as represented by a NO at the decision step S27, then control goes back to the step S25. Further, at a step S28, a plane function made by the four division polynomials of degree n is derived. Then, at a step S29, distortion correction coordinates of the target point are derived by using the plane function, and control goes back to the step S22, whereat the signal processing apparatus 500 is set to the standby mode (WAIT) until the next request signal is supplied.
The target points are pixels that are to be corrected in distortion by the signal processing unit 500. Since the distortion correction coordinates are different at respective points, all target points can be held by the register or can be obtained by using a variety of calculations whose circuit scales are larger than functions for use with lens.
However, the above-mentioned signal processing apparatus encounters with problems in which a memory capacity of a memory is large, real-time processing being difficult. Specifically, since the memory capacity is large, the scale of hardware is large. Hence, reduction of power consumption is difficult and the related-art signal processing apparatus is costly. Furthermore, the related-art signal processing apparatus cannot cope with the case in which characteristics of distortion of an image pickup lens are changed such as when the position of optical zoom is fluctuated or when a lens is exchanged.